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(54) Motion image indexing system and method 

(57) A video source (101), such as a VCR, plays 
back a motion image which is input by an image input 
section (111). 

Partition of scenes and creation of images repre- 
senting scenes are performed by a catalog creation 
section (120). 

Additionally, at this time, an image representing the 
movement of an object appearing in the scenes is cre- 
ated. A representative image and a motion description 



image are initially stored In a catalog storage section 
(130) and then displayed in catalog form by a display 
section (104) under control of a display control section 
(114). Since not only the representative image of 
scenes but also the movement of the object in those 
scenes are displayed, the contents of the scenes are 
easily grasped. 
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Description 

Background of the Invention 

Reld of the Invention 5 

The invention relates to a representative iniage dis* 
play method and apparatus comprising of a series of 
multiple images into which a motion image is formed, 
and which displays a representative image of each io 
scene, and to motion image search apparatus which 
execute searches for a motion Image using the repre- 
sentative image display apparatus. 

Description of Prior Art is 

Generally, a motion image contains multiple 
scenes. Each scene has independent meaning, and is 
defined by a group of consecutive images. Rg. 2 shows 
a situation where three scenes have been recorded on 20 
a video tape. Scene A is a view of a person, scene B is 
of a house, and scene C is of an automobile. Each 
scene respectively comprises two (A1 and A2), two (B1 
and B2), and three (C1-C3) images. A motion image 
search apparatus is used to quickly select a required 2s 
scene from a motion image on recorded media, such as 
video tapes, when editing motion image images such as 
those required in a multimedia presentation creation or 
in broadcasting. Recently, apparatus have been devel- 
oped that display a representative image (hereinafter 30 
refen'ed to as representative image) of each scene after 
a motion image has been partitioned into scene units, 
and that allow a desired scene to be selected using the 
representative image as an Index. 

Fig. 3 is a block diagram from a catalog for "Media- 3s 
chet/CUT by Hitachi, Ltd. showing a system using the 
product Details of \\s operation are given in "Media- 
chef/CUT for Windows User Operation Manual" (Win- 
dows is a registered trademark). 

In the figure, the system comprises a PC 2 which 40 
performs general control, a video playback apparatus 4 
whk;h provides the motion image to the PC 2, and a 
cobr printer 6 which prints the edited images. The PC 2 
contains a video capture board 8 which captures and 
digitizes images provided from a video playback appa- 4s 
ratus 4. This configuration perfonms, for example, the 
following steps. 

(1) First the motion image to be processed is 
loaded, (2) a transition point in the scene is automati- 
cally klentified based on the degree of color change in so 
the images. (3) an image is decided to be a representa- 
tive image that appears immediately after the transitbn 
point is observed, and (4) the representative image is 
stored or printed as an index for each scene. In the case 
of Fig. 2, a scene transition point is identified between ss 
A2 and B1, and between B2 and C1. and the leading 
images of each scene, B1 and CI. become the repre- 
sentative images. 

This system helps the user in tape editing, for 



example, by printing out the representative images con- 
tained in one video tape cassette in a smalt sheet and 
affixing the sheet to the side of the video tape cassette. 
In addition, editing processes such as sorting, deletion, 
and trimming of the scenes are possible on the PC 2. 

In the system descnbed above, referencing the rep- 
resentative images allows the user to look for desired 
scenes and replay them, that is, to search for a motion 
image. However, in representative images where a ship 
is afloat on the ocean, for example, the direction of the 
ship's movement can only be determined by replaying 
the motion image. If several representative images of 
this sort exist, for example, when searching for pictures 
where the ship moves to the right, the user must check 
each scene containing the representative image one by 
one. This manual search is not only laborious when 
there are a large number of similar representative 
images, but inconvenient when one tries to grasp the 
general movenfent of a motion image, and leaves much 
room for improvement for efficient editing. 

Summary of the Invention 

The inventton takes this problem into conskieration 
and provides a representative image display method 
and apparatus, and motion image search apparatus, 
that enable the user to easily and intuitively grasp the 
contents of a motion Image. 

The representative image display method of the 
invention includes a scene partition process whKh par- 
titions a motion image comprising a sequence of multi- 
ple images into scenes, a representative image 
selection process which selects a representative image 
with which to represent each partitioned scene, a transi- 
tion detection process which detects transitions occur- 
ring in the subject of each partitioned scene, a transition 
information hold process which associates and holds 
information concerning detected transitions for every 
scene with tiie representative image of the con-espond- 
ing scene, and a display process which combines and 
displays the representative image of at least one scene 
with transition information that was associated with the 
representative image. 

in this configuration, the motion image to be proc- 
essed is first input and then partitioned into scenes. 
Next, a representative image Is selected to represent 
each partitioned scene. Furtiiermore. at this time, 
changes occurring for an object within each scene are 
detected, and information relating to the detected tran- 
sitions for each scene is held in association witii the 
scene. Here "images" refer to elements which comprise 
a motion image, for example frames or MPEG pictures. 
Hereinafter, they will also be referred to as frames. An 
"object** means anything visible in the motk>n picture like 
a person. For example, in motion Image, characters 
appearing on screen can be the objects. A "changes 
occunring for an object" refers to, for example, a change 
due to movements of an object or a change in color of 
an object. 
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Next, the representative image of a scene and the 
transition inforniation associated with the representative 
image are combined and displayed. 

In accordance with the representative image dis- 
play method of the invention, the contents of a motion 
image can be easily and intuitively grasped since infor- 
mation concerning changes occurring in an object is 
displayed in addition to a representative image. 

On the other hand, the representative image dis- 
play apparatus of the Invention includes an Image input 
section which inputs and stores a motion image com- 
prising a sequence of multiple images, a scene partition 
section which analyzes the input motion image, detects 
scene transition points, and partitions the motion image 
into scenes, a representative image selection section 
which selects a proper image as the representative 
image from among the images comprising each scene, 
a change information hold section which detects 
changes occurring in an object within a scene, and 
associates and holds the information concerning tiie 
change lor every scene with the representative image of 
the corresponding scene, and a display control section 
which combines and displays a representative image of 
at least one scene with the change information that was 
associated with tiie representative image. 

In this configuration, tiie motion image which was 
input into the apparatus is first stored by the Image Input 
section. 

Next, the motion image Is read, its scene transition 
points detected, and it is partitioned into scenes. Then, 
both a representative image of each scene is aeated 
and the information on changes occurring in an object in 
every scene is associated and held witti tiie representa- 
tive image of the corresponding scene. Finally, a repre- 
sentative image of at least one scene and the change 
information that was associated with the representative 
image are combined and displayed. 

In accordance with the representative image dis- 
play apparatus of the invention, an effect similar to that 
of tiie above method is obtained. Furthermore, the 
apparatus can be easily incorporated into the motion 
Image search display apparatus. 

In one aspect of the representative image display 
apparatus of the invention, the changes occurring in the 
above object result from tiie movemerrt of the object, 
and the above change detection section detects the 
movement of tiie object based on differences among 
multiple images. Naturally, the "movement of the objecT 
may occur as a result of movement at the camera. A 
"difference" between images is determined, for exanv 
ple, by obtaining tiie difference in pixel values at corre- 
sponding locations. On this assumption, the apparatus 
further includes a motion description Image creation 
section which creates images visually showing move- 
ment of tiie object, and the above display control section 
combines and displays the above representative image 
with this motion description image. 

The apparatus enables tiie movement to be easily 
grasped in the object. 



In anotiier aspect the invention, the above motion 
description image aeation section fetches, at a preset 
time interval, an image stored in the image input section 
and aeates its outline image. The outline image is tiien 

5 superimposed orito a previously aeated motion 
description image so as to aeate a new motion descrip- 
tion image, and tiie fetch, the creation of tiie outline 
image and the superimposition processes are repeated 
to create n&n motion description images in succession. 

10 The "preset time interval" can be specified from an 
external source or can be the default value of tiie appa- 
ratus. In this aspect tiie outline images at the various 
times are all shown in one image. Only the outline 
image that was created first may be used without 

75 change as a motion description image. 

According to this aspect of tiie invention, it is even 
easier to grasp the movement. 

In yet another aspect of the invention, when creat- 
ing outline images at preset time intervals, tiie above 

20 motion description image creation section creates new 
motion desaiption images virhiie changing the colors of 
tiie outlines based on tfie elapsed time from a scene's 
start time. Here "color" includes blacK white, and neu- 
tral tints (or achromatic shades). 

25 In tills aspect, not only tiie movement but also tiie 
direction of movement can be easily grasped. 

In still another aspect of the invention, when an 
arbitrary time between the scene start time and end 
time is specified, tiie above display control section high- 

30 lights the location of the object at tiie specified time in 
tiie above motion description image. Since tiie "location 
of the subject" is indicated from the outline image, one 
application of the aspect is to change tiie intensity of 
outiine image that con-esponds to tiie specified time. 

35 According to tiiis aspect, it becomes easy to grasp 
tiie location of tiie object at a certain time, and this can 
fecititate the search for the search start position. 

The motion image search apparatus of the inven- 
tion performs searches for motion image and includes 

40 one of tiie above representative display apparatus, a 
video source which plays back and supplies a motion 
Image to the above representative display apparatus, a 
video source control section which controls the play- 
back of tiie motion image by tiie video source, and a 

45 command input section which inputs commands for 
search processing. In this configuration, when one 
scene and a playback start time from ttie displayed 
scenes are specified by the command, the vkieo source 
control section follows the command and performs play- 
so back of tiie motion image from tiie above playback start 
time of the specified scene. The playback enables ttie 
user to confirm the search result 

According to the motion image search apparatus of 
ttie invention, all the advantages of the above represent- 

55 ative image display apparatus are obtainable and tiie 
motion image editing efficiency is greatiy improved. 
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Brief Description of the Drawings 

Fig. 1 fs an overall block diagram of the nx)tion 
image search apparatus concerning embodiment 1. 

Fig. 2 exemplifies a state where three scenes have 
been recorded on video tapa 

Rg. 3 is a block diagram, taken from a catalog for 
Mediachef/CUT, of a system using "Mediachef/CUT. 

Fig. 4 Is a block diagram of motion description 
image creation section 1 23 of emlxxJiment 1 . 

Fig. 5 is a flowchart showing the flow of processes 
executed by the motion description creation sectiori 1 23 
of embodiment 1 . 

Fig. 6 shows a Ul which is displayed at display sec- 
tion 104 of embodiment 1. 

Fig. 7 is a flowchart showing the flow of processes 
which use the Ul of Fig. 6. 

Fig. 8 is an enlargement of the motion descriptbn 
image display area included in Fig. 6. 

Fig. 9 is a block diagram of a motion description 
image aeation section of embodiment 2. 

Fig. 1 0 is a f bwchart showing the flow of processes 
executed by the motion description image creation sec- 
tkm 123 of ^nbodiment 2. 

Rg. 11 shows a result obtained when the motion 
description image of Fig. 8 is recreated by the appara- 
tus of emtxxfiment 2. 

Fig. 12 Is a block diagram of a catalog creation sec- 
Xion 120 of embodiment 3. 

Fig. 1 3 is a block diagram of a catalog storage sec- 
tion 130 of embodiment 3. 

Fig. 14 shows the Ul which is displayed at display 
section 104 of embodiment 3. 

Description of Preferred Embodiments 

Here, preferred emt)odiments of a representative 
image cfisplay apparatus and a motion image search 
apparatus incorporating the representative image dis- 
play apparatus are described. 

This description will also clarify the representative 
Image display method of the invention. In the motton 
image search apparatus described in the embodiments 
below, a playback apparatus which plays back a video 
medium is included, and the playt>ack apparatus is elec- 
trically controlled directly from a main control section in 
the motion image search apparatus. Although this con- 
figuration Is ideal, for example, in broadcasting where 
relatively advanced functions are required, its applica- 
tons are of course not limited thereto. 

Embodiment 1 

In this embodiment, an apparatus is described 
which creates ah Image showing the motion of an object 
in every scene and displays it together with a represent- 
ative image. 



Configuration 

Rg. 1 is an overall block diagram of the motion 
image seairch apparatus concerning this eni^odiment. 

5 Since a main control section 110 controls the overall 
apparatus in this figure, this and a command input sec- 
tion 103 mentioned later correspond to tiie representa- 
tive Image display apparatus. A video source 101 plays 
back recorded video media such as those for VCRs and 

10 provides motion image to the main control section 110 
and a motion Image display section 102. The main con- 
trol section 110 creates representative images and the 
motion description images from ttie provided motion 
image, and motion image display section 102 simply 

15 displays tiie provided motion image. The motion Image 
display section 102. for example, a monitor which dis- 
plays ordinary NTSC format signals, allows a user to 
directiy view tiie searched motion image. 

A video source control section 112 controls the 

20 playt)ack of motion Image from a video source 101. It 
performs control functions required for the search proc- 
ess, such as "stop after 10 seconds from the start", in 
addition to the various control functions (tape rewind, 
fast fbnvafd, pause, and so fbrtii) available on the front 

25 panel of home VCRs. Control signals are connected to 
the video source 101 by a cable, A video source control 
section 1 12 controls the playt)ack of the motion image 
that is necessary for the main conf ol section 1 1 0 to cre- 
ate a representative Image, and necessary for the 

30 motion display section 1 02 to display. 

TTie played back motion image is input by an image 
input section 111. Image input section 1 1 1 includes an 
A/D converter and a frame memory (not shown). Image 
input section 1 1 1 digitizes the input Image, stores it into 

35 the frame memory and waits for further processing. The 
image input section 1 1 1 may also contain a disk storage 
apparatus which secondarily stores the contents of the 
frame menfX)ry. 

In this case, It becomes unnecessary to read the 

40 motion image required for processing every time from 
video source 101, and processing efficiency can be 
inrproved. 

A catatog creation section 120 is composed of a 
series of software modules which read image data from 

45 the frame memory, and creates a catalog based on this 
data, and which comprises a scene partition section 
121 , a representative Image selection section 122, and 
a motion description image aeation section 123. A cat- 
ak)g is a list of information concerning multiple scenes. 

so On the other hand, based on the contents of the data 
output from tiie catalog creation section 120. a catalog 
storage section 130 sorts and stores tiie data, and com- 
prises a scene partition data storage section 131 , a rep- 
resentative image storage section 132, and a motion 

55 description image storage section 1 33. 

The scene partition section 121 analyzes the digi- 
tized image using known methods and partitions the 
motion image into scenes based on the difference 
between two consecutive scenes. Based on tiie position 
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(for example a time code or counter value such as LTC 
or vrrc recaded on a tape) on the medium at the par- 
tition point, the start point and end point of each scene 
are stored in a scene partition data storage section 1 31 . 

The representative image selection section 122 
selects a certain frame from the scene as a representa* 
tive image. The representative image nnay be a franrre at 
the l^eginning, middle, or end of the scene, or a frame 
specified by the user. To identify the scene which the 
representative image represents, information such a 
scene number is stored into the representative image 
storage section 132. 

Motion descr^tion image creation section 123 per- 
forms a distinctive process in this embodiment Fig. 4 is 
a block diagram of a motion descrqation image creation 
section 123. 

In this figure, an outline image creation section 201 
processes the input image and creates an outline 
image. A composite image creation section 202 per- 
forms composite processing on the created outline 
image and an outline image storage section 203 tempo- 
rarily stores the composed outline image. In this embod- 
iment, the outline of an object included in a given image 
within a scene is sampled in each Image, and th^e out- 
lines are superimposed to create one image. 

Command input section 103 permits the entry of 
command inputs by the user and employs a device such 
as a keyix)ard, mouse, or various pointing devices. The 
input command is processed by an input management 
section 113, which is a command dispatcher, to initiate 
the required action within the main control section 1 10. 

A display control section 114 includes a system 
program for the display system which centrally controls 
the display of various images such as representative 
images and motion description images, and a display 
circuit such as a VGA controller and its driver. In the dis- 
play control section 114, the display state such as for 
the location of a catalog which is read from catalog stor- 
age section 130 is ultimately determined and displayed 
at a display section 104. The display section 104 may 
be an ordinary display apparatus on which representa- 
tive images and motion description images are dis- 
played. 

Process Operations 

The main control section 110. which receives a 
scene partition start instruction based on command 
inputs, issues a play-back instruction to the video 
source 101 through the vkleo source control section 
112. The motion image output from video source 101 is 
displayed by the motion image display section 102 and 
digitized in frame units by the image input section 111. 

Next, each frame is sent to a catak)g creation sec- 
tion 120. At tfie scene partition section 121. a histogram 
is taken for the input image and its preceding image, 
and a scene's partition point is judged when the stso- 
lute value of its difference reaches or exceeds a thresh- 
okl. This method itself is known. Instead of this method, 



any characteristic physical parameter between consec- 
utive images may be compared to detect the partition 
point. Once the partition point is ascertained, informa- 
tion such as a time code recorded on the video tape is 

5 acquired from tiie video source 101 and stored in the 
scene partition data storage section 131 . In parallel with 
this, a representative image is aeated and stored as a 
still image file in the representative image storage sec- 
tion 132. Next, the motion description image is created. 

10 Rg. 5 is a flowchart showing the flow of processes 
by motion description image creation section 123. 
According to this figure, the sent image is passed 
through a Compass-gradient filter in tiie outline image 
creation section 201 and an outline image is created 

15 (S21 1). In the outline image, each pixel takes on one of 
two values 0 and 1 , where pixels on the outiine become 
1 and other pixels become 0. Outiine images may be 
obtained at a user specified frame interval, for example, 
every 10 frames, or at an interval preset as a default 

20 value in the apparatus. 

Next, the image which has been stored in the out- 
line image storage section 203 is read and composed 

(5212) by the composite image creation section 202 
with the outline image Created in S21 1. Composing of 

25 the image is achieved by taking the logical OR of pixel 
values at the same coordinates for all pixels in the two 
images. The composite image thus aeated is stored 

(5213) in the outline image storage section 203. These 
processes are repeated until a scene partition point is 

30 detected. When the scene partitk)n point is detected, 
the image stored in the outiine image storage section 
203 is transferred to and stored in the motk>n descrip- 
tion image storage section 133. 

Here, the above processes are described together 

35 with user input operations and actual image display 
examples. 

Fig. 6 shows a user interface (Ul) displayed by the 
display section 104, and Fig. 7 is a flowchart showing 
the flow of processes using the Ul. In Rg. 6, a scene 

40 partition start button 152 starts scene partition. A play- 
back button 153 initiates playback of the motion image 
for the selected scene and a stop button 154 stops the 
motion image being played back. In Rg. 6, the entire 
screen above these buttons is a catalog display area 

45 151 which is organized into 3 scene display areas 155, 
each con^esponding to one scene. Each scene display 
area 155 contains a representative image display area 
156 which displays the representative image of that 
scene, a motk)n description image display area 157 

so which displays the motion description image, and a 
scene partition data display area 158 which displays 
each scene's start point (start time) and end point (end 
time). The first two digits of the time display indicate the 
hours, followed t}y minutes, seconds, and frame 

55 number. The frame number for the NTSC format is from 
0 to 29 since the number of frames displayed is 30 per 
second. When playing back a motion image, the user is 
allowed to specify beforehand how much to play back 
(that is, for how much of the motion image scene parti- 
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tion is to be performed). This specified amount is con- 
verted to a time code on the medium and held in the 
apparatus as the specified eti6 position. 

In the Ul. once the user presses the scene partition 
start button 152, the input management section 113 
interprets this command (S1 70] and a playback instruc- 
tion (SI 71) is sent to the video source 101 . The medium 
in the video source 101 is played back (SI 72) and the 
motion image is sent (Sl 73) to the catalog creation sec- 
tion 120. At this time, it is judged whether the time code 
on the medium indicates the specified end position 
(81 74) or the status indicates the end position (SI 75) of 
the medium, and if either is realized the process ends 
(SI 76). 

If the process does not end (No at SI 75). an outline 
image for that frame is created and a motion description 
image is created (SI 77). in parallel with this, the input 
image Is judged (Si 78) as to whether it should be a rep- 
resentative image, and a representative image is cre- 
ated (SI 79). for example, when a frame specified 
beforehand by the user is to be processed. 

Nesct. it is judged (SI 80) whether the scene parti- 
tion point is reached, and if it is. the motion description 
image is transfen-ed (SI 82) to the motion description 
image storage section 132. and catalog display Informa- 
tion is aeated (Si 83) to display (SI 84) scene partition 
data, the representative image, and the motion descrip- 
tion image. Then the process ends. If the scene parti- 
tion point is not reached, the next frame is read and the 
same process is repeated. 

From the steps above, a scene's representative 
Image and motion description image are displayed on a 
display apparatus. An enlarged diagram of the motion 
description image display area included in Rg. 6 is 
shown in Fig. 8. where the movement of the object 
extending over multiple images Is expressed consecu- 
tively in one image. As a result, the user can understand 
the scene In general from the representative image, and 
In detail from the motion description image. Based on 
these types of information, the user can actually view 
the scene by selecting it on the saeen, by using a 
mouse, for example. As shown in Fig. 6, the selected 
scene has its area frame highlighted. The above buttons 
become effective for the selected scene. That is, the 
playback button 1 53 enables playback of the scene and 
the stop button 154 enables playback to be stopped. In 
this manner, a highly efficient motion image search is 
achieved. The above is an overview of the eiribodimenL 
Possible improvements and variations for this enlbodt- 
ment are discussed in the following. 

(1) Display the motion description information as 
characters, symbols, and so forth. 

Instead of using a motion description image, 
the direction of movement of the subject can be 

expressed using characters such as "right" and 
"up", or using symbols such as and "t". This 
can be achieved by judging the angle of the sub- 
ject's movement vector. When expressing move- 



ment using characters, symbols, and so forth, the 
processing time and memory capacity required for 
the image aeation can be reduced. 
(2) Display the transition information other than 

5 nfX)vement 

When, for example, the object is a neon sign, 
the object itself does not ttme but Its color 
changes, in this case, examining such differences 
as color or brightness in the object before and after 

10 the change allow information such as "becomes 
red", l)ecomes white", or Tjecomes bright" to be 
displayed as text data beside the representative 
tmaga 

IS Embodiment 2 

Although the motion description image was created 
In embodiment 1 by superimposing successive outline 
Images, the superimposition In embodiment 2 is per- 
20 formed while varying the color of the outline image. For 
this reason, if the outline image concerning a subse- 
quently input frame is displayed using a darker color, for 
example, it becomes easier to grasp the movement of 
the display. 

25 Rg. 9 is a tHock diagram of the motion desalptlon 
image creation section 123 in this en*odlment. The dif- 
ference between this figure and Fig. 4 is that a hue set- 
ting section 301 is added. Hue setting section 301 
varies the hue of the outline image with the lapse of 

30 time. 

Rg. 1 0 is a flowchart showing the flow of processes 
executed by the motion description image creation sec- 
tion 123. The difference between this figure and Rg. 5 is 
that S312 is inserted after S21 1 . That is, after an outline 

35 image is created (S211). the outline's hue is varied 
(S312) by the hue setting section 301 with the lapse of 
time from the start time of the scene. After this, the proc- 
ess is repeated as shown in Rg. 5 and tiie motion 
description image is created. 

40 Gradation processing in which only the brightness 
is decreased can be employed In the processing of hue 
setting section 301 . In this case, the outline pixel values 
for the outiine image obtained the first time are set to a 
maximum value of 255 on an 8-blt gray scale and 

45 decreased by 5 at a time, for example, in subsequently 
input images. Fig. 1 1 is an image obtained as a result of 
re-creating the motion description image of Rg. 8 using 
tiie apparatus of this embocfiment and it becomes eas- 
ier to grasp the movement of the subject, 

so 

Embodiment 3 

in this embodimerrt, the input of an arbitrary time 
between the start time and end time of a scene high- 
55 lights the outiine image con'esponding to the Input time 
(hereinafter referred to as specified time) in the motion 
description image for the scene. Furthermore, tiie 
medium is actually fonwarded or reversed until the posi- 
tion where the image corresponding to the specified 
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time is found. 

Rg. 12 is a block diagram of a catalog creation sec- 
tion 120 in this embodiment and Rg. 13 is a Uock dia- 
gram of a catalog storage section 130. A new block 
added in Rg. 12 is a time calculation section 401 which 
obtains the amount of change in time occunring when a 
time button described later is pressed after the specified 
time was input for a scene. 

On the other hand, a new bk)ck In Rg. 13 is a time 
storage section 41 1 which stores, for every scene, the 
time calculated by the time calculation section 401 . The 
time storage section 411 is a buffer which can store at 
least one time value for every scene, and is useful when 
a search temporarily moves to another scene and then 
returns back to the original scene, so that the user can 
determine which image had been referenced pres^iously. 

Rg. 14 shows the Ul displayed by the display sec- 
tion 104. 

The difference between this figure and Fig. 6 is that 
a display of "current time" is newly added. When a spec- 
ified time is input, the current time is initialized with this 
value. Subsequently, every time a time button is 
pressed, the current time increases or decreases. Time 
buttons conprise a previous time button 451 which 
specifies a previous time and a subsequent time button 
452 which specifies a subsequent time. 

Using the configuration above, the user first selects 
a scene. Here, the topmost scene is selected. Next, the 
specified time is entered, for example, through a key- 
board. 

In the figure, the atlowat)le input is in the range of 
scene start time 00:00:00:00 to end time 00:02:05:20. 
For example, when 00:01 :02:28 is specified, the outline 
image with the closest time is highlighted. Images are 
input in 1/30 second units, and when an outline image is 
created for every Image, for example, the time corre- 
sponding to the outline image (called an outline image 
time) takes on a discrete value in 1/1 5 second units. For 
this reason, the specified time and the outline image 
time usually do not match exactly, so the closest time is 
selected. The selected outline image time is stored in a 
time storage section 41 1. 

In this embodiment, along with highlighting the out- 
line image, the playback position of the actual medium 
is forwarded or rewound to the outline image time. Thus, 
the object on the medium appears at the position of the 
highlighted outline image. As a result, the user can see 
where the object is located at the specified time, and if 
playback from that position is desired, this can be 
achieved simply by issuing a playback instruction. 

When the position at the specified time is different 
from where playback is to start, the above time buttons 
are pressed. The pressing of the button is judged by the 
input management section 1 13 and a new current time 
is calculated by the time calculation section 401 . Since 
the outline image time takes on a disaete value in 1/15 
second units, the current time increases or decreases 
by 1/1 5 second each time the button is pressed, and the 
outline image adjacent to the currently highlighted out- 



line image is highlighted. The playback positkm on the 
medium is also precisely adjusted. 

The contents of time storage section 411 are 
updated with the new cun-ent time. From the processes 

5 above, it becomes possik)le to set the start position of 
playback with extremely high precision. If the subse- 
quent time button 452 is pressed at the final outline 
image time of the scene, or conversely, if the previous 
time button 451 is pressed at the initial outiine image 

10 time, these instructions will be ignored. 

In this embodiment, it is necessary to consider that 
the motion desaiption image is only one image. That is, 
once this image is created, it becomes necessary to 
add information to the motion description image regard- 

15 ing which outline conresponds to what time. For this rea- 
son, (1) motion desCTiption images having differing 
levels of brightness according to preset times are cre- 
ated using the method of embodiment 2. (2) the corre- 
spondence between the various levels of brightness 

20 and times is estak)lished, (3) the pixels having the bright- 
ness corresponding to the current time are scanned in 
tiie motion description image, and (4) all matching pix- 
els are highlighted. The result of these steps is that only 
tiie desired outline stands out. 

2S While there have been described what are at 
present considered to be preferred embodiments of tiie 
invention, it will be understood that various modifica- 
tions may be made tiiereto. and it is intended that tiie 
appended claims cover all such nrxxJifications as fall 

30 within the true spirit and scope of the invention. 

Clainos 

1 . A representative innage display method comprising: 

35. 

a scene partition process for partitioning into 
scenes a motion image comprising a sequence 
of multiple images, 

a representative image selection process for 
40 selecting a representative image to represent 

each partitioned scene, 

a change detection process for detecting 
changes occurring in an object within each par- 
titioned scene. 

45 a change information hold process for associ- 

ating and holding information related to tiie 
detected changes for each scene witii the rep- 
resentative image of the scene, and 
a display process for combining and displaying, 

50 for at least one scene, the representative 

image of the scene with its associated change 
information. 

2. A representative image display apparatus compris- 
55 ing: 

an image input section (1 1 1) for inputting and 
storing a motion image comprising a sequence 
of multiple images. 
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a scene partition section (121) for analyzing 
the input motion image, detecting transition 
points in the scenes, and partitioning the 
motion image into scenes, 
a representative image selection section (1 22) s 
for selecting a proper image from the motion 
image comprising multiple scenes and setting it 
as a r^esentative image, 
a change information hold section (120, 130) 
for detecting changes occun^ing in an object in io 
each scene and associating and holding the 
tnfbrmatiori related to the change for each 
scene with the representative image of the 
scene, and 

a display control section (114) for comtsintng is 
and displaying, for at least one scene, the rep- 
resentative image of the scene with its associ- 
ated change infornfiation. 

3. The representative image display apparatus 20 
according to claim 2, wherein changes occurring in 

the object are caused by movement of the object, 
and the change information hold section (120, 130) 
detects the movement of the object based on a dif- 
ference among a plurality of images, and the dis- 25 
play control section (114) displays text which 
indicates the movement of the object included in 
the images when it displays the representative 
image. 

30 

4. The representative image display apparatus 
according to claim 2. wherein changes occurring in 
the object are caused by movement of the object, 
and the change information hold section (120. 130) 
detects the movement of the object based on a dif- 3S 
ference among a plurality of images, and further 
includes a motion description Image creation sec- 
tion (123) which aeates an image visually showing 

the movement of the object, and the display control 
section (1 1 4) combines and displays the represent- 40 
ative image and the motion description image. 

5. The representative image display apparatus 
according to claim 4, wherein said motion descrip- 
tion image creation section (123) fetches, at a pre- 4S 
set time interval, the image stored in the image 
input section and creates its outline image, and 
superimposes the outline image onto a previously 
created nfx>tion description image to repeatedly cre- 
ate a new motion description image. so 



according to claims 4 tiirough 6. wherein said dis- 
play control section (114) highlights the position of 
an ot)ject in a motion description image at a speci- 
fied time, when an arbitrary time is specified 
between the start time and end time of a scene. 

8. The representative image display apparatus 
according to claim 7, wherein said display control 
section (114) scans for pixels having a color core- 
sponding to the given time, and highlights the loca- 
tion of the ol3ject in a motion description image by 
changing the color of all the pixels found by scan- 
ning. 

9. The representative image display apparatus 
according to claim 2, wherein changes occun^ing in 
the object are changes in color of the object, and 
the change information hold section (120, 130) 
detects a change in color of the object based on dif- 
ferences among a plurality of images and the dis- 
play conti-ol section displays text which indicates 
the change in color of the object when said display 
control section (114) the representative image is 
displayed. 

10. A motion image search apparatus for performing 
searches for a motion image, comprising: 

the representative image display apparatus 
according to claims 2 through 9, 
a video source (101) for playing back and sup- 
plying a rTK>tion image to said representative 
image display apparatus, 
a video source control section (1 12) for control- 
ling the playback of the motion image from tiie 
video source, and 

a command input section (103) for inputting 
commands for search processing, 

wherein the video source control section 
(112) performs, when one scene from dis- 
played scenes and a playback start time for the 
scene are specified 1:^ a command, playback of 
the motion image from the playback start time 
of the scene based on the command. 



6. The representative image display apparatus 
according to claim 5, wherein said motion descrip- 
tion image creation section (123) creates new 
motion description images while changing colors of ss 
the outlines based on the elapsed time from tiie 
start time of the scene. 

7. The representative image display apparatus 
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